<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
  <meta http-equiv="Content-Style-Type" content="text/css" />
  <meta name="generator" content="pandoc" />
  <title>nfs挂载参数</title>
  <style type="text/css">code{white-space: pre;}</style>
  <style type="text/css">
div.sourceCode { overflow-x: auto; }
table.sourceCode, tr.sourceCode, td.lineNumbers, td.sourceCode {
  margin: 0; padding: 0; vertical-align: baseline; border: none; }
table.sourceCode { width: 100%; line-height: 100%; }
td.lineNumbers { text-align: right; padding-right: 4px; padding-left: 4px; color: #aaaaaa; border-right: 1px solid #aaaaaa; }
td.sourceCode { padding-left: 5px; }
code > span.kw { color: #007020; font-weight: bold; } /* Keyword */
code > span.dt { color: #902000; } /* DataType */
code > span.dv { color: #40a070; } /* DecVal */
code > span.bn { color: #40a070; } /* BaseN */
code > span.fl { color: #40a070; } /* Float */
code > span.ch { color: #4070a0; } /* Char */
code > span.st { color: #4070a0; } /* String */
code > span.co { color: #60a0b0; font-style: italic; } /* Comment */
code > span.ot { color: #007020; } /* Other */
code > span.al { color: #ff0000; font-weight: bold; } /* Alert */
code > span.fu { color: #06287e; } /* Function */
code > span.er { color: #ff0000; font-weight: bold; } /* Error */
code > span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */
code > span.cn { color: #880000; } /* Constant */
code > span.sc { color: #4070a0; } /* SpecialChar */
code > span.vs { color: #4070a0; } /* VerbatimString */
code > span.ss { color: #bb6688; } /* SpecialString */
code > span.im { } /* Import */
code > span.va { color: #19177c; } /* Variable */
code > span.cf { color: #007020; font-weight: bold; } /* ControlFlow */
code > span.op { color: #666666; } /* Operator */
code > span.bu { } /* BuiltIn */
code > span.ex { } /* Extension */
code > span.pp { color: #bc7a00; } /* Preprocessor */
code > span.at { color: #7d9029; } /* Attribute */
code > span.do { color: #ba2121; font-style: italic; } /* Documentation */
code > span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */
code > span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */
code > span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */
  </style>
  <link rel="stylesheet" href="doc/linux-style.css" type="text/css" />
</head>
<body>
<div id="header">
<h1 class="title">nfs挂载参数</h1>
</div>
<div id="TOC">
<ul>
<li><a href="#nfs挂载参数">nfs挂载参数</a></li>
<li><a href="#代码分析">代码分析</a></li>
<li><a href="#实际验证">实际验证</a></li>
</ul>
</div>
<h1 id="nfs挂载参数">nfs挂载参数</h1>
<p><code>mount -t nfs -o vers=4.2,timeo=600,retrans=3,proto=tcp,lock,soft  localhost:/s_test /mnt</code></p>
<p><code>mount -t nfs -o vers=4.2,timeo=600,retrans=3,proto=tcp,lock,hard  localhost:/s_test /mnt</code></p>
<p>从代码层面分析上面的挂载参数 hard 和soft的打印信息，以及定时器重传的规律，例如每次重传的时间，timeo的时间设置等等。</p>
<p>mount | grep nfs #查看mount 挂载参数</p>
<p>dmesg | less #查看日志信息</p>
<h1 id="代码分析">代码分析</h1>
<p>执行bpftrace查看这个函数的调用栈:</p>
<p><code>sudo bpftrace -e 'kprobe:xprt_set_retrans_timeout_rtt { printf(&quot;Caller: %s\n&quot;, kstack); }'</code></p>
<p>发现执行的时候，好像并没有执行xprt_set_retrans_timeout_rtt这个函数？</p>
<p>查看call_timeout调用栈：</p>
<pre><code>Caller: 
        call_timeout+1
        rpc_async_schedule+18
        process_one_work+418
        worker_thread+83
        kthread+262
        ret_from_fork+31</code></pre>
<p>查看xprt_adjust_timeout调用栈：</p>
<pre><code>Caller:
    xprt_adjust_timeout+1
    __rpc_execute+124
    rpc_async_schedule+18
    process_one_work+418
    worker_thread+83
    kthread+262
    ret_from_fork+31</code></pre>
<p>分析代码以内核4.19为例：</p>
<div class="sourceCode"><pre class="sourceCode cpp"><code class="sourceCode cpp">nf<span class="va">s_init_timeout_values</span>
    <span class="cf">switch</span> (proto) 
    <span class="cf">case</span> XPRT_TRANSPORT_TCP:<span class="co">//TCP用线性增长</span>
        to-&gt;to_increment = timeo * HZ / <span class="dv">10</span>;  <span class="co">// 默认使用初始值作为增量</span>
        to-&gt;to_exponential = <span class="dv">0</span>;  <span class="co">// 禁用指数退避</span>
    <span class="cf">case</span>: XPRT_TRANSPORT_UDP   <span class="co">// UDP 协议使用指数退避</span>
        to-&gt;to_exponential = <span class="dv">1</span>;;  <span class="co">//  使用用指数退避</span>

    rpc_async_schedule
        __rpc_execute
            xprt_adjust_timeout
                <span class="cf">if</span> (to-&gt;to_exponential)    <span class="co">//判断是否设置为指数增长</span>
                    req-&gt;rq_timeout &lt;&lt;= <span class="dv">1</span>;  <span class="co">// 指数退避：超时时间翻倍</span>
                <span class="cf">else</span>
                    req-&gt;rq_timeout += to-&gt;to_increment;  <span class="co">// 线性增长：增加固定增量</span>

            __rpc_add_timer
                <span class="co">//打印: &quot;RPC: %5u setting alarm for %u ms&quot;</span>
                rpc_set_queue_timer(queue, task-&gt;u.tk_wait.expires); <span class="co">//添加定时器到队列里面</span>
                list_add(&amp;task-&gt;u.tk_wait.timer_list, &amp;queue-&gt;timer_list.list);<span class="co">//添加任务到任务链表 </span>

    call_timeout
        <span class="cf">if</span> (RPC_IS_SOFT(task)) <span class="co">//判断是否为参数是否为soft</span>
        <span class="co">//打印: &quot;not responding, still trying或not responding, timed out&quot;</span></code></pre></div>
<p>也就是说在这段代码中，首先根据挂载参数判断是否是-tcp,</p>
<p>如果是tcp 则to-&gt;to_increment = timeo * HZ / 10; // 默认使用初始值作为increment增量</p>
<p>to-&gt;to_exponential = 0; // 禁用指数退避</p>
<p>也就是这这个增量值就使用默认初始timeo作为增量，比如你timeo =600, 那么每次增加的增量也是600， 然后不使用指数增长，即设置to-&gt;to_exponential = 0；</p>
<p>通过if (to-&gt;to_exponential) 判断是采用线性增长还是指数增长。</p>
<p>是线性增长：req-&gt;rq_timeout += to-&gt;to_increment;</p>
<h1 id="实际验证">实际验证</h1>
<p>客户端发起写操作 echo something &gt; /mnt/file 会触发nfs4_open</p>
<p>由于修改了内核代码,当收到 OPEN 请求时，服务器进入无限循环,已实现超时重传的效果</p>
<div class="sourceCode"><pre class="sourceCode cpp"><code class="sourceCode cpp">--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
<span class="er">@@</span> <span class="dv">-46</span>,<span class="dv">6</span> <span class="dv">+46</span>,<span class="dv">7</span> <span class="er">@@</span>
 <span class="pp">#include </span><span class="im">&quot;acl.h&quot;</span>
 <span class="pp">#include </span><span class="im">&quot;pnfs.h&quot;</span>
 <span class="pp">#include </span><span class="im">&quot;trace.h&quot;</span>
+<span class="er">#</span>include &lt;<span class="ot">linux</span>/delay.h&gt;
 
 <span class="pp">#ifdef CONFIG_NFSD_V4_SECURITY_LABEL</span>
 <span class="pp">#include </span><span class="im">&lt;linux/security.h&gt;</span>
<span class="er">@@</span> <span class="dv">-356</span>,<span class="dv">6</span> <span class="dv">+356</span>,<span class="dv">11</span> <span class="er">@@</span> nfsd4_open(<span class="kw">struct</span> svc_rqst *rqstp, <span class="kw">struct</span> nfsd4_compound_state *cstate,
        <span class="kw">struct</span> nfsd_net *nn = net_generic(net, nfsd_net_id);
        <span class="dt">bool</span> reclaim = <span class="kw">false</span>;
 
+       <span class="cf">while</span> (<span class="dv">1</span>) {
+               printk(<span class="st">&quot;</span><span class="sc">%s</span><span class="st">:</span><span class="sc">%d</span><span class="st">, sleep</span><span class="sc">\n</span><span class="st">&quot;</span>, <span class="ot">__func__</span>, <span class="ot">__LINE__</span>);
+               msleep(<span class="dv">20</span> * <span class="dv">1000</span>);
+       }
+
        dprintk(<span class="st">&quot;NFSD: nfsd4_open filename </span><span class="sc">%.*s</span><span class="st"> op_openowner %p</span><span class="sc">\n</span><span class="st">&quot;</span>,
                (<span class="dt">int</span>)open-&gt;op_fname.len, open-&gt;op_fname.data,
                open-&gt;op_openowner);</code></pre></div>
<p>结果：服务器永远不回复 OPEN 响应，客户端阻塞在 OPEN 阶段</p>
<p>挂载 并且打开调试开关</p>
<div class="sourceCode"><pre class="sourceCode sh"><code class="sourceCode bash"><span class="fu">mount</span> -t nfs -o vers=4.2,timeo=600,retrans=3,proto=tcp,lock,hard  localhost:/s_test /mnt
<span class="bu">echo</span> 0xFFFF <span class="op">&gt;</span> /proc/sys/sunrpc/nfs_debug <span class="co"># NFSDBG_ALL</span>
<span class="bu">echo</span> 0x7fff <span class="op">&gt;</span> /proc/sys/sunrpc/rpc_debug <span class="co"># RPCDBG_ALL</span>
<span class="bu">echo</span> somthing <span class="op">&gt;</span> /mnt/file <span class="kw">&amp;</span>  #<span class="ex">nfs</span>写入</code></pre></div>
<p>mount -t nfs -o vers=3,timeo=600,retrans=3,proto=tcp,lock,hard localhost:/tmp/s_test /mnt</p>
<p>通过 <code>dmesg | less</code> 查看日志：</p>
<div class="sourceCode"><pre class="sourceCode sh"><code class="sourceCode bash"><span class="bu">[</span>  204.970305] RPC:    50 setting alarm for 60000 ms
[  267.374370] RPC:    50 setting alarm for 120000 ms
[  390.277365] RPC:    50 setting alarm for 180000 ms
[  570.475071] nfs: server localhost not responding, still trying</code></pre></div>
<p>可以看到打印的时间，每次重传的时间可以刚好对应上。</p>
<p>然后根据挂载参数timeo=600,retrans=3 来分析一下<code>setting alarm for xxxx ms</code></p>
<p>基础超时时间为60000ms，第一次重传时间为60000 + 60000ms，第二次重传时间为 120000 + 60000 ms,</p>
<p>第三次重传未出现？可能是当时没捕捉到，再去验证一下。</p>
</body>
</html>
